Skip to content

Conversation

JeremyDahlgren
Copy link
Contributor

Follow up to work done in #131894 to make RemoteClusterService multi-project aware. This task plumbs the ProjectResolver down into the RemoteClusterService from NodeConstruction.

Resolves: ES-12572

Follow up to work done in elastic#131894 to make RemoteClusterService
multi-project aware.  This task plumbs the ProjectResolver down
into the RemoteClusterService from NodeConstruction.

Resolves: ES-12572
@JeremyDahlgren JeremyDahlgren added :Distributed Coordination/Network Http and internode communication implementations >refactoring Team:Distributed Coordination Meta label for Distributed Coordination team v9.2.0 labels Aug 15, 2025
@JeremyDahlgren
Copy link
Contributor Author

JeremyDahlgren commented Aug 18, 2025

There are elasticsearch multi-project tests that are not using the default ProjectResolver, that at query time are resolving to an actual origin project, while the remote cluster setup in these tests are still using the ClusterSettings based linked project configuration that currently does not support multi-project. So these tests will not work with this PR until the ProjectCustom based linked project configuration mechanism is in place that supports multi-project. The commit used here to plumb the ProjectResolver may have to be included as part of that work, the only issue though is the new linked project configuration mechanism using ProjectCustom would be added to serverless, not to the elasticsearch codebase.

Example test:

./gradlew --debug ":x-pack:qa:multi-project:core-rest-tests-with-multiple-projects:yamlRestTest" --tests "org.elasticsearch.multiproject.test.CoreWithMultipleProjectsClientYamlTestSuiteIT" -Dtests.method="test {yaml=cluster.stats/30_ccs_stats/cross-cluster search stats search}"

Test config:

...
---
"cross-cluster search stats search":
  - requires:
      test_runner_features: [ capabilities ]
      capabilities:
        - method: GET
          path: /_cluster/stats
          capabilities:
            - "ccs-stats"
      reason: "Capability required to run test"

  - do:
      cluster.state: {}
  - set: { master_node: master }
  - do:
      nodes.info:
        metric: [ http, transport ]
  - set: {nodes.$master.http.publish_address: host}
  - set: {nodes.$master.transport.publish_address: transport_host}

  - do:
      cluster.put_settings:
        body:
          persistent:
            cluster:
              remote:
                cluster_one:
                  seeds:
                    - "${transport_host}"
                  skip_unavailable: true
                cluster_two:
                  seeds:
                    - "${transport_host}"
                  skip_unavailable: false
  - is_true: persistent.cluster.remote.cluster_one
...
[2025-08-25T11:44:37,894][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [test-cluster-1] fatal error in thread [elasticsearch[test-cluster-1][clusterApplierService#updateTask][T#1]], exiting java.lang.AssertionError: The default project ID should not be used in multi-project environment
          at [email protected]/org.elasticsearch.transport.RemoteClusterService.getConnectionsMapForProject(RemoteClusterService.java:732)
          at [email protected]/org.elasticsearch.transport.RemoteClusterService.updateRemoteCluster(RemoteClusterService.java:485)
          at [email protected]/org.elasticsearch.transport.RemoteClusterService.updateRemoteCluster(RemoteClusterService.java:430)
          at [email protected]/org.elasticsearch.transport.RemoteClusterAware.validateAndUpdateRemoteCluster(RemoteClusterAware.java:219)
          at [email protected]/org.elasticsearch.common.settings.AbstractScopedSettings$2.apply(AbstractScopedSettings.java:350)
          at [email protected]/org.elasticsearch.common.settings.AbstractScopedSettings$2.apply(AbstractScopedSettings.java:322)
          at [email protected]/org.elasticsearch.common.settings.AbstractScopedSettings$SettingUpdater.lambda$updater$0(AbstractScopedSettings.java:664)
          at [email protected]/org.elasticsearch.common.settings.AbstractScopedSettings.applySettings(AbstractScopedSettings.java:175)
          at [email protected]/org.elasticsearch.cluster.service.ClusterApplierService.applyChanges(ClusterApplierService.java:526)
          at [email protected]/org.elasticsearch.cluster.service.ClusterApplierService.runTask(ClusterApplierService.java:460)
          at [email protected]/org.elasticsearch.cluster.service.ClusterApplierService$UpdateTask.run(ClusterApplierService.java:159)

Copy link
Member

@ywangd ywangd left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For the YAML test failures, the short answer is that we can mute them in the associated build.gradle file. Please also add a comment to explain why the tests need to be muted.

The issue is that settings update consumers are invoked in a context without project. How to notify components for per-project setting changes is still an open issue (ES-11463). We might be able to fix the tests once the issue is resolved. Technically it may be out of scope for CPS if it only supports config via ProjectCustom. In that case, these tests will simply be incompatible with multi-project. If the same test coverage is critical, we (the ES team not just distrib-coord) will need to add a version that uses the ProjectCustom so that it can be tested in MP setup on the serverless side.

Comment on lines 258 to 259
// NOTE: Only for use in tests
public TransportService(
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: Convention for this kind of comment is something like "public for testing"

@JeremyDahlgren
Copy link
Contributor Author

The issue is that settings update consumers are invoked in a context without project. How to notify components for per-project setting changes is still an open issue (ES-11463). We might be able to fix the tests once the issue is resolved. Technically it may be out of scope for CPS if it only supports config via ProjectCustom. In that case, these tests will simply be incompatible with multi-project. If the same test coverage is critical, we (the ES team not just distrib-coord) will need to add a version that uses the ProjectCustom so that it can be tested in MP setup on the serverless side.

I was running into another test failure (see below). Is the assertion too strict given the current state of the codebase in terms of multi-project support? In 69a704b I replaced the assertion with a @FixForMultiProject annotation to add the assertion that the default project ID should not be used in multi-project environments, resulting in a clean CI run. Is this sufficient for now until the rest of the codebase and test infrastructure is further along in the multi-project work?

./gradlew ":x-pack:qa:multi-project:xpack-rest-tests-with-multiple-projects:yamlRestTest" --tests "org.elasticsearch.multiproject.test.XpackWithMultipleProjectsClientYamlTestSuiteIT"
...
[2025-08-27T23:46:10,675][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [yamlRestTest-0] fatal error in thread [elasticsearch[yamlRestTest-0][ml_utility][T#1]], exiting java.lang.AssertionError: The default project ID should not be used in multi-project environment
        at [email protected]/org.elasticsearch.transport.RemoteClusterService.getConnectionsMapForProject(RemoteClusterService.java:685)
        at [email protected]/org.elasticsearch.transport.RemoteClusterService.getConnectionsMapForCurrentProject(RemoteClusterService.java:677)
        at [email protected]/org.elasticsearch.transport.RemoteClusterService.getRegisteredRemoteClusterNames(RemoteClusterService.java:189)
        at [email protected]/org.elasticsearch.transport.RemoteClusterService.groupIndices(RemoteClusterService.java:176)
        at [email protected]/org.elasticsearch.action.ResolvedIndices.resolveWithIndexNamesAndOptions(ResolvedIndices.java:172)
        at [email protected]/org.elasticsearch.action.ResolvedIndices.resolveWithIndicesRequest(ResolvedIndices.java:154)
        at [email protected]/org.elasticsearch.action.search.TransportSearchAction.executeRequest(TransportSearchAction.java:365)
        at [email protected]/org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:336)
        at [email protected]/org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:126)
        at [email protected]/org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:135)
        at [email protected]/org.elasticsearch.action.support.ActionFilter$Simple.apply(ActionFilter.java:54)
        at [email protected]/org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:132)
        at [email protected]/org.elasticsearch.action.support.MappedActionFilters$MappedFilterChain.proceed(MappedActionFilters.java:71)
        at [email protected]/org.elasticsearch.action.support.MappedActionFilters.apply(MappedActionFilters.java:49)
        at [email protected]/org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:132)
        at [email protected]/org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$4(SecurityActionFilter.java:182)
        at [email protected]/org.elasticsearch.action.ActionListenerImplementations$DelegatingFailureActionListener.onResponse(ActionListenerImplementations.java:233)
        at [email protected]/org.elasticsearch.xpack.security.authz.AuthorizationService.runRequestInterceptors(AuthorizationService.java:671)
        at [email protected]/org.elasticsearch.xpack.security.authz.AuthorizationService.handleIndexActionAuthorizationResult(AuthorizationService.java:642)
        at [email protected]/org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$14(AuthorizationService.java:539)
        at [email protected]/org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:1062)
        at [email protected]/org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:1028)
        at [email protected]/org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:33)
        at [email protected]/org.elasticsearch.action.support.SubscribableListener$SuccessResult.complete(SubscribableListener.java:406)
        at [email protected]/org.elasticsearch.action.support.SubscribableListener.tryComplete(SubscribableListener.java:326)
        at [email protected]/org.elasticsearch.action.support.SubscribableListener.addListener(SubscribableListener.java:222)
        at [email protected]/org.elasticsearch.action.support.SubscribableListener.addListener(SubscribableListener.java:180)
        at [email protected]/org.elasticsearch.xpack.security.authz.AuthorizationService.authorizeAction(AuthorizationService.java:536)
        at [email protected]/org.elasticsearch.xpack.security.authz.AuthorizationService.maybeAuthorizeRunAs(AuthorizationService.java:450)
        at [email protected]/org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorize$3(AuthorizationService.java:337)
        at [email protected]/org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:258)
        at [email protected]/org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:33)
        at [email protected]/org.elasticsearch.xpack.security.authz.RBACEngine.lambda$resolveAuthorizationInfo$0(RBACEngine.java:176)
        at [email protected]/org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:261)
        at [email protected]/org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.lambda$getRoles$4(CompositeRolesStore.java:216)
        at [email protected]/org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:261)
        at [email protected]/org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.getRole(CompositeRolesStore.java:224)
        at [email protected]/org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.getRoles(CompositeRolesStore.java:209)
        at [email protected]/org.elasticsearch.xpack.security.authz.RBACEngine.resolveAuthorizationInfo(RBACEngine.java:172)
        at [email protected]/org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:353)
        at [email protected]/org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$5(SecurityActionFilter.java:178)
        at [email protected]/org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:261)
        at [email protected]/org.elasticsearch.action.ActionListenerImplementations$MappedActionListener.onResponse(ActionListenerImplementations.java:111)
        at [email protected]/org.elasticsearch.xpack.security.authc.AuthenticatorChain.authenticate(AuthenticatorChain.java:100)
        at [email protected]/org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:274)
        at [email protected]/org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:178)
        at [email protected]/org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.applyInternal(SecurityActionFilter.java:174)
        at [email protected]/org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$apply$1(SecurityActionFilter.java:115)
        at [email protected]/org.elasticsearch.xpack.core.security.SecurityContext.executeAsInternalUser(SecurityContext.java:170)
        at [email protected]/org.elasticsearch.xpack.security.authz.AuthorizationUtils.switchUserBasedOnActionOriginAndExecute(AuthorizationUtils.java:172)
        at [email protected]/org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:111)
        at [email protected]/org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:132)
        at [email protected]/org.elasticsearch.action.support.TransportAction.handleExecution(TransportAction.java:96)
        at [email protected]/org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:59)
        at [email protected]/org.elasticsearch.tasks.TaskManager.registerAndExecute(TaskManager.java:222)
        at [email protected]/org.elasticsearch.client.internal.node.NodeClient.executeLocally(NodeClient.java:107)
        at [email protected]/org.elasticsearch.client.internal.node.NodeClient.doExecute(NodeClient.java:85)
        at [email protected]/org.elasticsearch.client.internal.support.AbstractClient.execute(AbstractClient.java:160)
        at [email protected]/org.elasticsearch.client.internal.support.AbstractClient.search(AbstractClient.java:295)
        at [email protected]/org.elasticsearch.xpack.core.ClientHelper.executeAsyncWithOrigin(ClientHelper.java:224)
        at [email protected]/org.elasticsearch.xpack.ml.datafeed.persistence.DatafeedConfigProvider.expandDatafeedConfigs(DatafeedConfigProvider.java:507)
        at [email protected]/org.elasticsearch.xpack.ml.datafeed.DatafeedConfigAutoUpdater.runUpdate(DatafeedConfigAutoUpdater.java:80)
        at [email protected]/org.elasticsearch.xpack.ml.MlAutoUpdateService.runUpdate(MlAutoUpdateService.java:76)
        at [email protected]/org.elasticsearch.xpack.ml.MlAutoUpdateService.lambda$clusterChanged$4(MlAutoUpdateService.java:70)

@JeremyDahlgren JeremyDahlgren marked this pull request as ready for review August 27, 2025 22:41
@JeremyDahlgren JeremyDahlgren requested a review from a team as a code owner August 27, 2025 22:41
@elasticsearchmachine
Copy link
Collaborator

Pinging @elastic/es-distributed-coordination (Team:Distributed Coordination)

@ywangd
Copy link
Member

ywangd commented Aug 28, 2025

OK that's unfortunate but seems to be the best course of action right now. The reason is that ML code is larget not MP ready. More specifically, it should a projectClient instead of a client here.

Copy link
Member

@ywangd ywangd left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@JeremyDahlgren JeremyDahlgren merged commit feb99ea into elastic:main Aug 28, 2025
33 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

:Distributed Coordination/Network Http and internode communication implementations >refactoring Team:Distributed Coordination Meta label for Distributed Coordination team v9.2.0

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants